HOPE is a second-order pruning method for Mixture-of-Experts models that removes experts while accounting for cooperative interactions between them. It targets the 95% of memory in large MoEs dedicated to idle experts, offering a static alternative to dynamic offloading. The method scores experts based on pairwise dependencies rather than independent scalar values.
It is worth trying if you are running large MoE models where static memory constraints matter more than dynamic loading overhead. The catch is that it requires a calibration dataset and does not eliminate the need for quantization to fit models in memory. It is not for those who need zero-shot pruning without any calibration data.
more: https://huggingface.co/blog/amazon/higher-order-pruning-experts
Leave a Reply